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THE RELATIONSHIP OF CERTAIN COURSE CHARACTERISTICS TO 
STUDENT RATINGS OF SCIENCE AND MATHEMATICS TEACHING 
AT FOUR-YEAR AND TWO-YEAR COLLEGES 

Maurice W. Villano 
The Pennsylvania State University 

Background 

The purpose of this paper was to report the effect of selected course 
variables on the outcome of student instructional ratings at two 4-year 
colleges and at seventeen 2-year branch colleges or learning centers of a 
major state university where the College of Science had systematically 
evaluated all science and mathematics teaching of approximately 15,000 
students in almost 500 classes during a recent academic term. The specific 
aspects of teaching performance being rated were described in the context 
of relevant course characteristics such as class enrollment, course level, 
instructor rank, type or location of campus, and major subject or disci- 
pline area. Multivariate statistical analyses were used to explain the 
complex interrelationships. Results were interpreted with a view toward 
sustaining or rejecting the sometimes popular notion that ascribes the 
highest student ratings to classes with so-called "favorable** course 
characteristics. Implications for the evaluation of science and mathe- 
matics teaching were discussed, including the use of controls for **bias** 
in ratings, revision of the instrument, and possible future areas of study. 

Reviews of the literature by Costin, Greenough & Menges (1971), McKeachie 
(1973) and others have found the extent to which student course ratings are 
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influenced by instructional variables generally to be inconclusive or even 
contradictory. Research in this area has continued, however, as witnessed 
by sor>e of the -nore recent validation studies.' Aleamoni & Graham (1974) 
investigated the effect of class size, course level, and Instructor rank 
on Course Evaluation Questionnaire (CEQ) results at the University of 
Illinois. Bejar 5. Doyle (1974) noted that factor structures of student 
ratings were generalizable over those attributes that separated main campus 
students from those at academic centers. Miller (1974) discussed the im- 
Tjact of subject matter on research findings. In one of the few studies in 
science teaching, Cornwall (1974) reported the differential effects of 
instructional variables on chemistry courses rated at Wisconsin. 

Despite the lack of convincing support for their position, many faculty 
members view the highest course ratings as being assigned to small classes 
of upper-division courses taught by senior, ranks at main university campuses. 
Persistence of this notion has hindered full acceptance of student evaluation 
systems by profe<)sors who expect rating devices to measure what is being 
realistically perceived by students within the context of relevant instruc- 
tional variables. Many institutions control for possible "bias" introduced 
by these variables by providing separate norming of evaluation results. 
There is increasing evidence, however, that student ratings vary also as a 
function of the specific facets of teaching performance being evaluated, 
and that these complex interrelationships can best be studied using large 
data samples and a multivariate statis'^ical analysis approach. 

Research on instructor and course ratings in science is especially 
meager. Further studies are required to improve student evaluation systems 
with their important implications for education at the university level. 



Procedures 

The research instrument used id this invest" igation was a local adaptation 
of the Student Appraisal of Teaching (SAOT) fom by R. I. Miller (1972) (see 
Appendix). This rating device has 15 Likert-type item-statements to which 
subjects respond on a four-step scale. The response data were collected on 
over 18,000 students in undergraduate and graduate science and mathematics 
classes at two 4-year colleges (106 classes including the main university 
campus) and 17 two-year colleges or learning centers (382 classes) of a 
maior state educational institution during the Winter 1974 academic term. 
The sample v;as screened for missing item responses on individual SAOT forms 
and reduced to 14,366 Ss representing a total of 488 class-sized groups. 

The response data described above were processed and the item means for 
each class were computed. The generated 15 by 15 intercorrelation (R) matrix 
representing betwecn-instroctot covariation wa^ submitted to a principal-^ 
components analysis (Hotelling, 1933) with unities inserted in the diagonal. 
By slightly over-factoring (using an eigenvalue cut-off of 0.96 as the root 
criterion), three psychologically-meaningful components were extracted and 
subjected to an orthogonal (Varimax) rotation (Kaiser, 1958). Component 
scores were calculated for the three dimensions of the SAOT that were 
descriptive of teaching performance using the direct measurement method: 

X ^ (F'F)"^ F' Z 

where X = factor (component) scores, (F*F)"^ * inverse of the original item 
R matrix, F* « orthogonally-rotated facto- loading matrix, and Z « standard 
scores (Hannan, 1967) . 



These < ata (component scores) were then treated as dependent variables in 
a2x2x2x2x3 completely-crossed factorial design with two levels each 
for fixed factors of Size (class enrollment of 1-20 and 21 & above), Level 
(introductory and advanced courses) , Rank (instructor & assistant professor 
.and associate & full professor), Campus (four-year colleges and two-year 
learning centers), and three levels of subject area or Discipline (biologi- 
cal, physical, and mathematical sciences). Multivariate analysis of variance 
(MANOVA) was conducted to examine the main effects and interactions of the 
five factors. Step-down F tests were employed to indicate the dependent 
variables that contributed most to the multivariate F ratios. Multiple 
discriminant function analysis was used to assess the extent and direction 
of significant effects and interactions. The foregoing procedures were exe- 
cuted using the SUNY (Buffalo) MULTIVARIANCE Program (Finn, 1968). 

A possible limitation of this study from the methodological standpoint 
was the arbitrary collapsing of several levels of the factors in the MANOVA 
design to only two or three levels. VThile this was based, in part, on an 
a priori analysis of the data, some information may have been lost, thus 
affecting the results. There were some missing cells or cells with only one 
observation; consequently, the means of the null subclasses were estimated 
and the residual sum of squares and the within sum of squares were not 
equal. The three-way and higher-level interactions were not tested. Sev- 
eral inestimable sets of contrasts for the higher-level interactions wei 
removed from the model and these interaction sums of squares were pooled 
with the residual to form the error term, with the degrees of freedom ad- 
justed accordingly. 



Results and Conclusions 



The principal-components analysis yielded three distinct, independent, 



and over 77-percent of the total instrument variance (see Table 4) • These 
factors, descriptive of the teaching performance being rated, were labelled, 
with their respective Coefficient Alpha reliabilities, as follows: 
Component I — Instructional Qualities (•96); Component II — Thinking- 
Participation (.84); and Component III — Instructor Fairness (.88). 

The factor array comprising the tirst component (see Table 1) consisted 
of seven items that appeared to relate to general instructional qualities of 
organization, clarity, motivation, and interest. The final instructor * 
rating and ''recommending the course to a friend" also helped define this 
factor indicating that all seven items in the aggregate were inseparable 
from the overall evaluation itself. 

The second component (Table 2), encompassing three items only, addressed 
itself to the encouragement of thinking, participation, and discussion in 
the classroom. The third and last component (Table 3), composed of five 
items, was indicative of instructor interaction with the students regarding 
fairness in grading, rendering assistance, and other classroom relationships. 

The M\NOVA disclosed statistically significant main effects for all 
five factors of Size , Level , Rank , ' Campus , and Discipline and for three 2- 
way interactions: Size by Level , Size by Rank , and Level by Rank . The 
discriminant functions separated cells Xvrith high and low mean scores on the 
dependent variables into identifiable groups which took on special meaning 
in this study. An extended discussion of the interpretations follows. 




and robust dimensions underlying the SAOT that accounted for all of the items 
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For the Size main effect, the Thinking/Participation variable was solely 
responsible for the highly significant (.0001) multivariate F ratio^(T»ble 
6). The discriminant function for this effect (Table 8) indicated that 
ratings high on Thinking/Participation separated the large-sized classes, 
* highest on this function, from the small-sized classes. 

For the Level main effect, the Thinking/Participation variable again 
accounted singularly for the multivariate F (.02). The discriminant 
function for this effect signif;led that evaluations high on Thinking/ 
Participation discriminated between the lower-level courses, highest on 
this function, from the upper-level courses. 

For the Rank main effect, both the Instructional Qualities and the 
Instructor Fairness variables contributed significantly to the multivariate 
F ratio (.01). The discriminant function for this effect suggested that 
low ratings on Instructional Qualities and Instructor Fairness differen- 
tiated between Junior-ranked faculty, highest on this function, and senior- 
ranked faculty. 

For the C ampus main effect, the Thinking/Participation variable, ^dnce 
more, most individually influenced the multivariate F (.03). The function 
for this effect revealed that high ratings on Thinking/Pa rticipation f:«nded 
to distinguish the four-year colleges, highest on this function, from the 
tev-year colleges or learning centers. 

For the Discipline or subject area main effect, both the Thinking / 
Participation and Instructional Qualities variables explained the hig>»ly 
significant F (.0001). The function for this effect noted that high ratings 
on these variables somewhat disassociate the biological sciences, highest 
on this function, from the physical and mathematical sciences. 
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For the Size by Level interaction, the Instructor Fairness variable was 
mainly responsible for the significant (.01) ou.itivariate F ratio (Table 7). 
The discriminant function for this interaction indicated that low ratings 
on Instructor Fairness separated the snail-sized upper-level courses, highest 
on this function, from the large-sized lower- level courses. 

For the Size by Rank Interaction, the Thinking/Participation variable 
primarily accounted for the multivariate F ratio (.04). The discriminant 
function for this interaction showed thatJiigb_ratings on Thinking/Participation 
and to a lesser extent Instructor Fairness tended to differentiate between 
large-sized classes taught by senior faculty, highest on this function, and 
small-sized classes taugjht by Junior faculty. 

For the Level by Rank interaction, the Instructor Fairness variable 
contributed most to the multivariate F ratio (.03). The discriminant function 
for this interaction was based on a low rating on Instructor Fairness and to 
a lesser extent Thinking/Participation and appeared to distinguish between 
upper-level courses taught by junior faculty, highest on this function, from 
lower-level courses taught by senior faculty jnerabors. 

Based on the above findings, it was concluded that the hypothesis that 
assigned the highest student evaluations to courses with "ideal" characteris- 
tics was not entirely supported. The results were sonteuhat equivocal in 
that the means for small-sized classes were less thaa for large-sized classes, 
and were less for upper-level than lower-level courses; hoxvever, senior faculty 
members fared better than Junior ones, and ratings at the four-year institu- 
tions exceeded those at the two-year colleges and learning centers. The two- 
way interactions and discriminant functions underscored these relationships 
in a multivariate sotti.i;for most instances. 
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Prom an inspection of the cell mean scores following the multiple discri- 
minant function analysis (Table 5), a vague but discemable pattern emerged 
that tended to separate the class-sized groups into two gross evaluation 
categories: 1) high ratings—large-sized, lower-level biological science 
courses taught by senior faculty at the four-year colleges; and 2) low ratings— 
small-sized, upper-level physical and mathematical science courses taught by 
junior faculty members at the two-year colleges and learning centers. While 
this dichotomy is not always clearcut— there are several anomolies — it points 
to. a need for the separate norming of evaluation results if they are to be 
meaningful* 

The inordinate influence of the Thinking/Participation variable on three 

main effects and one interaction wa? puzzling. This component, consisting 

j 

of only three items and accounting jfor 22-percent of the variance for the SAOT, 

i 

comprises a minor part of many evaluation forms. The role of critical 
thinking in science evaluation may be u|dere8timated and should bear further 
investigation in future studies. 

It was recommended that the Student Appraisal of Teaching ( SAOT) instru- 
ment be retained in its present form pending further study. Such investi- 
gation should include but not be limited to an exploration of the factorial 
invariance or stability of the teaching performance aspects ^uaderlying the 
SAOT and an examination of changes in instructor scores for succeeding terms 
along the sane dimensions brought out in the present study. 
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APPFNDfX 



STUDENT APPRAISAL OF TEACHING 



Thoughtful student reaction can help improve teaching effectiveness. This 
questionnaire is designed for that purpose. Your assistance is appreciated. 
Please jdo not sign vour name. 

Rate your professor on each item by indicating the extent of your agreement 
with the statement. 

If you conclude that you cannot give an informed answer, please indicate 
"Cctnnot answer" (E) . All students should answer item 15. 



A 

Strongly 
agree 



Agree 



Disagree 



' D 

Strongly 
disagree 



. CannjDt 
answer 



1. Cl^ss presentations are well organized. 

2. Mastery of the subject matter is evident. 

3. Critical thinking and analysis are encouraged. 

4. Examinations that require creative, original 
thinking are giv^n. 

5. The presentation motivates vour best effort. 

6. Ideas presented are clearly explained. 

7. The professor accepts and is willing to discuss 
student viewpoints different from his own. 

8. YoM are encouraged to seek the professor's help 
when needed. 

9. Grading policies are adequately explained and 
followed. 

10. The professor exhibits a genuine interest and 
enthusiasm in teaching. 

11. Questions on examinations are fair. 

12. The ptofessor encourages student participation 
and welcomes questions and discussion. 

13. Learning objectives for the course and/or 
examinations are clearly stated. 

14. I would recoTTunend this instructor to a friend. 

15. Compared to other professors you have had at this 
university, how effective is the professor in this 
course (considering the previous points)? 



A 
A 
A 
A 



B 
B 
B 
B 



C 
C 
C 
C 



D 
D 
D 
D 



A B C D 

A B C D 

A B C D 

A B C D 

A B C D 

A B C D 

A B C D 

A B C D 

A B C D 

A B C D 

A B C D 



E 
E 
E 
E 

E 
E 
E 

E 

E 

E 

E 
E 



E 
E 
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One of the most 
effective 



B 

More effective 
than most 



Not as effective 
as most 
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One of the least 
effective 



TABLE 1 

COMPONENT I: INSTRUCTIONAL QU/\LITIES 



Item No* 


Loading 


SAOT Statement 


: 1 


• 86 


Class presentations are well'-organized. 


6 


.76 


Ideas presented are c plained* 


15 


• 75 


Compared to other prof^-sscrs you have had at this 
university, how effective is the professor in 
this course (considering the previous points)? 


2 


• 71 


Mastery of the subject matter is evident. 


14 


.67 


I would recommend this instructor to a friend. 


5 


• 67 


The presentation motivates your best effort, 


10 


• 55 


The profep?or exhibits a genuine interest and 
enthusiasm in teaching* 



TABLE 2 

COMPONENT II: THINKING-PARTICIPATION 



Item No, 


Loading 


SAOT Statement 


3 


• 85 


Critical thinking and analysis are encouraged • 


4 


• 82 


Examinations that require creative, origi**-! 
' thinking are given • 


12 


• 64 


The professor encourages student participation 
and welcomes questions and answers • 
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TABLE 3 

COMP(»IENT III: INSTRUCTOR FAIRNESS 



Item No. Loading ^AOT Statement 



9 .79 Grading policies are adequately explained and 

followed* 

IX »72 Questions on examinations are fair. 

8 ,62 You are encouraged to seek the professor's help 

when needed. 

7 ,62 The professor accepts and is willing to discuss, 

student viewpoints different from his own. 

13 .62 Learning objectives for the course and/or 

examinations are clearly stated. 
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TABLE 4 

COMPONENT LOADING MATRIX USING ORTHOGONAL TRANSFORMATION* 



Item Instructional Thinking- Instructor 

No. Qualities (I) Participation (II) Fairness (III) 

1 ^ 86 

6 . 76 47 
15 75 47 

2 71 40 

14 67 57 

5 67 41 46 

10^ 55 47 44 

3 85 

4 82 

12 64 54 

9 79 
11 72 

8 53 62 

7 53 62 
13 57 62 

Total variance 

accounted for: 29.3% 22% 25. 9Z 



Loadings multiplied by 100 and rescaled to eliminate negative signs. Coef 
ficients less than .40 deleted. 
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TABLE S 

CELL MEANS (Z SCORES) FOR THREE DEPENMNT VARIABLES 





"Size 


Levi 


Rank 


Camp 


Disc 


1 


sml 


lo 


Jr 


4 


phys 


1 


sml 


lo 


Jr 


4 


macn 


15 


sml 


lo 


Jr 


2 


oiol 


31 


sol 


lo 


Jr 


2 


phys 


86 


sml 


lo 


Jr 


2 


math 


2 


sml 


lo 


sr 


4 


math 


4 


sml 


lo 


sr 


2 


DlOl 


15 


sml 


lo 


sr 


2 


phys 


2 


sml 


lo 


sr 


2 


matti 


4 


sml 


up 


Jr 


4 


DlOl 


2 


sml 


up 


Jr 


4 


phys 


9 


sml 


up 


Jr 


4 


matn 


3 


sml 


up 


Jr 


2 


blol 


25 


sml 


up 


Jr 


2 


phys 


10 


sml 


up 


Jr 


2 


math 


6 


sml 


up 


sr 


4 


blol 


5 


sml 


up 


sr 


4 


phys 


3 


sml 


up 


sr 


4 


matn 


2 


sml 


up 


sr 




phys 


4 


Irg 


lo 


Jr 


4 


blol 


10 


Irg 


lo 


Jr 


4 


phys' 


12 


Irg 


lo 


Jr 


/ 
*f 


math 


37 


Irg 


lo 


Jr 


o 


blol 


47 


Irg 


lo 


Jr 


I 


phys 


58 


Irg 


lo 


Jr 


Z 


math 


8 


Irg 


lo 


sr 


/. 
*l 


W4 ^1 

blol 


7 


Irg 


lo 


sr 


4 


phys 


4 


Irg 


lo 


sr 




macn 


6 


Irg 


lo 


sr 


2 


blol 


13 


Irg 


lo 


sr 


2 


phys 


10 


Irg 


lo 


sr 


2 


math 


3 


Irg 


up 


Jr 


4 


blol 


4 


Irg 


up 


Jr 


4 


phys 


6 


Irg 


up 


Jr 


4 


math 


5 


Irg 


up 


Jr 


2 


phys 


3 


Irg 


up 


Jr 


2 


math 


4 


Irg 


up 


sr 


4 


blol 


10 


Irg 


up 


sr 


4 


phys 


1 


ire 


up 


sr 


4 


math 



^-mutDbec of sections 



Inst Qual Instr Fair Think/Part 



0.590 


0.081 


0.224 


0.581 


-0.875 


0.224 


0.369 


0.604 


0.499 


-0.128 


0.386 


-0.267 


-0.166 


-0.035 


-0.004 


-0.316 


0.507 


-0.701 


0.535 


-0.297 


-0.779 


0.153 


0.162 


-0.802 


0.119 


-1.109 


0.320 


-0.505 


-0.403 


-0.718 


-2.242 


-0.557 


1.562 


0.740 


-0.411 


-0.548 


0.266 


-0.369 


-1.535 


-0.300 


0.065 


-0.740 


-0.163 


-0.785 


-0.207 


0.519 


-0.064 


0.445 


0.569 


0.210 


-0.656 


-0.579 


0.219 


-1.095 


-1.649 


0.305 


-0.977 


0.245 


-0.089 


1.050 


0.110 


0.260 


0.698 


-0.010 


-0.386 


-0.147 


0.461 


-0.149 


0.628 


-0.238 


-0.220 


-0.023 


-0.133 


-0.138 


0.090 


0.286 


0.139 


0.986 


-0.211 


0.530 


0.267 


0.113 


0.228 


0.654 


-0.013 


0.554 


0.508 


0.181 


-0.057 


0.069 


0.540 


-0.349 


-0.178 


0.319 


-0.397 


-0.063 


-0.475 


0.207 


-0.308 


-0.465 


0.315 


-0.305 


0.215 


0.?42 


-6.865 


0.441 


0.067 


1.096 


1.004 


-b.t)02 


0.903 


0.403 


1.083 


0.247 


0.314 


3.598 


-1.762 



, / 
per cell. ^See p. 4 for definition of factors & levels. 



TABLE 6 

TESTS OF SIQIIFICANCE: MAIN EFFECTS 



Dependent Variable 



Univariate F Step-down F Multivariate F 



SIZE 

Instructional Qualities 
Instructor Fairness 
Thinking/Participation 



(dfh-1, dfe-467) 
1,82 1.82 
0.26 0.23 
25.91**** 26.47**** 



(dfnunr3. dfden'^^S) 

F - 9.54 
p<.0001 



LEVEL 

Instructional Qualities 
Instructor Fairness 
Thinking/Participation 



RANK 

Instructional Qualities 
Instructor Fairness 
Thin k ing /Participation 



(dfh*l, dfe-467) 
0.07 0.07 
0.29 ' 0.29 
10.52** 10.36** 



(dfnunf3, dfden=^65) 

F - 3.58 
p<.02 



(dfh-1, dfe-467) 
5.68* 5.68* 
5.89* 6.06* 
0.12 0.03 



dfden-465) 

F - 3.94 
p<.01 



CAMPUS 
Instructional Qualities 
Instructor Fairness 
Thinking/Participation 



DISCIPLINE 
Instructional Qualities 
Instructor Fairness. 
Thinking/Participation 



(dfh-1, dfe-467) 
0.27 0.27 
0.94 0.96 
7.80** 8.01** 



(dfnuin-3, dfden-^65) 

F - 3.09 
P<.03 



(dfh-2. dfe-467) 
7.03*** 7.03*** 
2.80 2.76 
10.55**** 11.15**** 



(dfnum-^» dfden=930) 

F - 7.00 
p<. 0001 



*p<.05, **p<.01, ***p^.001, ****p<.0001 
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TABLE 7 



TESTS 


OF SIGNIFICANCE: 


INTERACTIONS 




Dependent Variable 


Univariate F 


Step-down F 


Multivariate F 


<?T7F bv LEVEL 


^ n ' 


dfp-467) 


(dfnu„-3, dfden"^65) 


Instructional Qualities 


1.02 


1.02 


F - 5.37 


Instructor Fairness 


13.95*** 


14.05*** 


p<.01 


Thinking/Participation 


0.83 


1.01 




<;T7^ hv RANK 


(dfu"l. 


dfe-467) 


<dfniim"3» <i^den-^65) 


Instructional Qualities 


O.OA 


0.04 


F - 2.82 


Instructor Fairness 


2.17 


2.18 


p<.04 


Thinking/Participation 


' 6.08* 


6.20* 




LEVEL by RANK 


(dfh-1. 


dfe»467) 


(df„u«-3, dfden-«65) 


Instructional Qualities 


0.07 


0.07 


F - 3.10 


Instructor Faimv^iss 


5.35* 


5.37* 


p<.03 


Thinking/Participation 


3.69 


3.83 





*p<.05, **p<.01, ***p<-.001 



TABLE 8 

STANDARDIZED DISCRIMINANT COEFFICIENTS 



Main Effect/ 
Interaction 



Bartlett 
Chl-Sq 



Dependent Variable 
Instructional Instructor Thinking/ 
Qualities Fairness Participation 



SIZE 



27.81 
df-3, p<.0001 



0.30 



-0.08 



0.97 



LEVEL 



10.62 
df-3, p<.02 



-0.03 



-0.15 



0.99 



RANK 



11.67 
df»3, p^r.Ol 



-0.70 



-0.72 



0'.05 

\ 



CAMPUS 



9.18 
df-3, P4.03 



0.23 



0.33 



0.93 



DISCIPLINE 



41.14 
df-6, p^.OOOl 



0.66 



-0.08 



0.79 



SIZE by LEVEL 



15.85 
df-3, P<.01 



-0.29 



-0.94 -0.25 



SIZE by RANK 



LEVEL by RANK 



8.38 
df«3, p<.04 



9.22 
df-3, p<.03 




0.13 



-0.14 



0.52 



0.86 



-0.77 -0.65 
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